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DETAILED ACTION 



1. Claims 33-47 and 49-61 are pending. 

2. Applicant's arguments, see regarding a single camera performing the calibration, filed 
1 1/25/2009, with respect to the rejection(s) of claim(s) 33-47 and 49-61 under under 35 
U.S.C. 102(b) as being anticipated by Wei et al. (Multisensory Visual servoing by a Neural 
Network, 1999) have been fully considered and are persuasive. Therefore, the rejection has been 
withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

4. This application currently names joint inventors. In considering patentability of the 
claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of the various 
claims was commonly owned at the time any inventions covered therein were made absent any 
evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1 .56 to point out 
the inventor and invention dates of each claim that was not commonly owned at the time a later 
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invention was made in order for the examiner to consider the applicability of 35 U.S. C. 103(c) 
and potential 35 U.S. C. 102(e), (f) or (g) prior art under 35 U.S. C. 103(a). 

5. Claims 33-38, 40-47 and 49-61 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over McGee et al. (US 4942539 A) in view of Wei et al. (Multisensory Visual servoing by a 
Neural Network, 1999). 

As per claim 33, McGee et al. teaches a single camera for automatically determining the 
position and orientation of an object by utilizing as few as a single digital image generated by as 
few as a single camera without the use of a structured light. The digital image contains at least 
three non-colinear geometric features of the object that equates to the single camera of this 
application (see abs. and fig. 1 , particularly the single camera and "a programmed computer 
together with reference data and camera calibration data to provide at least three non-parallel 3-D 
lines. The 3-D lines are utilized by an iterative algorithm to obtain data relating to the position 
and orientation of the object in 3-D space."). McGee et al. does not specifically teach the 
following limitations as taught by the secondary reference Wei et al. 

Wei et al, in the other hand teaches a multisensory visual servoing by a neural network, 
including a capturing a number of images of a calibration object (see fig. 4) by the camera 
(see page 279, col. 2, lines 1-7, and fig. 4, wherein evidence has shown about two camera instead 
of one); determining a set of (intrinsic parameters) 1 (see page 276, col. 2, section II., and col. 1, 



Intrinsic parameters = The intrinsic matrix containing 5 intrinsic parameters. These parameters encompass focal length, 
image format, and principal point. The parameters a x =f m x and a, =/ ■ my represent focal length in terms of pixels, where m x 
and m y are the scale factors relating pixels to distance. Nonlinear intrinsic parameters such as lens distortion are also important 
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second paragraph, wherein both intrinsic and extrinsic parameters have been covered by 
"sensor/camera calibration", and fig. 2 for motion parameters) of the camera (see figs. 2-3) from 
at least one of the number of images of the calibration object (see page 276, col. 1, third 
paragraph, lines and 1-8, and page 279, col. 2, lines 12-13) captured by the camera (as seen in 
fig. 3 below); and determining a set of extrinsic parameters (see page 1, col. 1, second paragraph, 
wherein both intrinsic and extrinsic parameters have been covered) of the camera (see figs. 2-3) 
from at least one of the number of images of the calibration object captured by the camera (see 
fig. 4, wherein the clearer picture has been considered as calibrated), the set of extrinsic 
parameters (sec page 1, col. 1, second paragraph, wherein both intrinsic and extrinsic parameters 
have been covered as noted above) comprising a camera (see figs. 2-3, wherein any one of the 
cameras performs the required function, and having more than one falls under design choice) 
space-to-training space transformation defining a transformation between a space reference 
frame and a training space reference frame (see page 2, particularly "space-to-training space 
transformation" has been interpreted as from object to the camera/range finders, to the training 
and to the geometric transformation, "a space reference frame" has been interpreted as from the 
reference position to the algorithm module to the camera/range finders, to the shift and finally to 
the geometric transformation and col. 1, wherein reference frame has been considered as position 
frame). 



although they cannot be included in the linear camera model described by the intrinsic parameter matrix. Man} modem camera 
calibration algorithms estimate these intrinsic parameters as well. 
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As per claim 34, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a positioning the camera (see figs. 2-3) with respect to the calibration object (see page 
276, second paragraph lines 1-5 and col. 2, section II., particularly "Here uniqueness means that 
the sensory data should be able to uniquely determine the end-effector's relative position with 
respect to the object, while redundancy requires that one use more than the minimum number of 
sensory data to achieve robustness in the pose determination."). 

As per claim 35, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a method wherein positioning the camera (see figs. 2-3) with respect to the calibration 
object (see abs. and fig. 4) comprises positioning the camera orthogonally (see fig. 3, wherein the 
camera has been considered mounted orthogonally) with respect to a ruled template with a 
number of features (see page 5, table II, which has been considered as template), where a known 
or determinable physical relationship exists between at least some of the features (see page 277, 
cols. 1-2, particularly table I, wherein roll, pitch and yaw have been considered as physical 
features). 
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As per claim 36, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a method wherein positioning the camera (see figs. 2-3) with respect to the calibration 
object (see abs. and fig. 4) comprises positioning the camera (see figs. 2-3, wherein the camera 
being position to see/track an object on a conveyor) with respect to a sample of a type of object 
the robot will manipulate (see fig. 2, particularly the object and page 276, section I. lines 1-5), 
the sample having a number of features, where a known or determinable physical relationship 
exists between at least some of the features (see page 277, cols. 1-2, particularly table I, wherein 
roll, pitch and yaw have been considered as physical features). 

As per claim 37, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a method wherein capturing a number of images of a calibration object (see abs. and 
fig. 4) by the camera comprises capturing at least one image at each of a plurality of positions 
spaced perpendicularly from the calibration object (see page 276 col. 1 , third paragraph, 
particularly "used an eye-on-hand configuration to track an object on a conveyor." being on the 
conveyor brings a plurality of positions even perpendicular position). 

As per claim 38, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a method wherein capturing a number of images of a calibration object by the camera 
(see figs. 2-3) comprises capturing at least one image at each of a plurality of different 
orientations with respect to the calibration object (see page 276 col. 1, third paragraph, 
particularly "used an eye-on-hand configuration to track an object on a conveyor." being on the 
conveyor brings a plurality of positions even different orientations). 
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As per claim 40, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a method wherein determining a set of extrinsic parameters (see page 276, col. 2, 
section II., and col. 1, second paragraph, wherein both intrinsic and extrinsic parameters have 
been covered by "sensor/camera calibration", and fig. 2 for motion parameters) of the camera 
(see figs. 2-3 as noted above) from at least one of the number of images of the calibration object 
(see fig. 4) captured by the camera, the set of extrinsic parameters (see page 276, col. II, section 
II., and col. 1, second paragraph, wherein both intrinsic and extrinsic parameters have been 
covered by "sensor/camera calibration", and fig. 2 for motion parameters) comprising a camera 
(see figs. 2-3) spacc-to-training space transformation defining a transformation between a camera 
space reference frame and a training space reference frame (see page 2, particularly "space-to- 
training space transformation" has been interpreted as from object to the camera/range finders, to 
the training and to the geometric transformation, "a space reference frame" has been interpreted 
as from the reference position to the algorithm module to the camera/range finders, to the shift 
and finally to the geometric transformation and col. 1 , wherein reference frame has been 
considered as position frame) comprises determining a respective translation component along 
three orthogonal axes, and a respective rotation component about the three orthogonal axes (see 
fig. 2, wherein the axes the box between the shift and <P, has been considered as orthogonal 
axes). 

As per claim 41, Wei et al. , teaches a multisensory visual servoing by a neural network 
including a method that further comprising determining a camera space-to-tool space 
transformation based (see page 276, col. 1, particularly "tracking an object on a conveyor" has 
automatically determine camera space-to-tool and page 277, section B. Dealing with the 
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Inexactness of the Mapping has dealt with the transformation) at least in part on at least two of 
the number of images captured by the camera of the calibration object (see fig. 4, has been 
broadly interpreted as the two images of the calibration). 

As per claim 42, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a method that further comprising: determining a camera space-to-tool space 
transformation based (see page 276, col. 1, particularly "tracking an object on a conveyor" has 
automatically determine camera space-to-tool and page 277, section B. Dealing with the 
Inexactness of the Mapping has dealt with the transformation) on single one of the number of 
images captured by the camera of the calibration object (sec fig. 3 as noted above) and on a 
number of physical coordinates of at least one feature of the calibration object (see page 277, 
cols. 1-2, particularly table I, wherein roll, pitch and yaw have been considered as physical 
features). 

As per claim 43, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a method that further comprising: capturing an image of a teaching object (see fig. 2, 
wherein the object has been considered as a teaching one) of a type of object that will be 
manipulated by the robot (see abs. and fig. 3); selecting a number of features from the captured 
image of the teaching object (see abs., and fig. 4); determining a set of object space coordinates 
for each of the selected features from the captured image of the teaching object (see fig. 4). 

As per claim 44, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a method wherein selecting a number of features (see page 2, col. 1) from the captured 
image of the teaching object (see fig. 4, and the object on the conveyor as noted above being 
teaching object to be captured) comprises selecting six features from the captured image of the 
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teaching object (see page 280, particularly table II, has been considered having a plurality of 
selectable image position). 

As per claim 45, Wei et al. , teaches a multisensory visual servoing by a neural network 
including a method that further comprising determining an object space-to-camera space 
transformation defining a transformation between an object space reference frame and the 
camera space reference frame (see page 2, particularly "space-to-training space transformation" 
has been interpreted as from object to the camera/range finders, to the training and to the 
geometric transformation, "a space reference frame" has been interpreted as from the reference 
position to the algorithm module to the camera/range finders, to the shift and finally to the 
geometric transformation and col. 1, wherein reference frame has been considered as position 
frame). 

As per claim 46, Wei et al. . teaches a multisensory visual servoing by a neural network 
including a method that comprising determining a position and an orientation of an object frame 
in the tool frame reference frame (see page 276, col. 1 , wherein reference frame has been 
considered as position frame and table II. for position and orientation) based at least in part on 
the object frame-to-camera space and camera space-to-tool space transformations (see abstract, 
pages 1-2 and fig. 2). 

As per claim 47, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a method that further comprising: providing the position and orientation of the object 
(see fig. 4) frame to the robot; and training an intended operation path inside the object frame 
(see page 276 col. 1, third paragraph, particularly "used an eye-on-hand configuration to track an 
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object on a conveyor." being on the conveyor brings a plurality of positions even different 
orientations). 

As per claim 49, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a method that further comprising: adjusting a position of the movable portion of the 
robot if the number of features located in the captured image of the target object is determined to 
be an insufficient number of features (see page 276, col. 1 , particularly the abs. for calibration); 
and capturing another two-dimensional image of the volume containing the target object (see fig. 
2-4, wherein image taken by the camera being considered has having the capacity to do two- or 
three-dimensional images) before determining the object spacc-to-camera space transformation 
for the target object (see figs. 2 and 3 as noted above). 

As per claim 50, McGee et al. teaches a single camera for automatically determining the 
position and orientation of an object by utilizing as few as a single digital image generated by as 
few as a single camera without the use of a structured light. The digital image contains at least 
three non-colinear geometric features of the object that equates to the single camera of this 
application (see abs. and fig. 1 as see below, particularly the single camera and "a programmed 
computer together with reference data and camera calibration data to provide at least three non- 
parallel 3-D lines. The 3-D lines are utilized by an iterative algorithm to obtain data relating to 
the position and orientation of the object in 3-D space."). McGee et al. does not specifically 
teach the following limitations as taught by the secondary reference Wei et al. 
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Wei et al., in the other hand teaches a multisensory visual 
servoing by a neural network, including capturing a two-dimensional image of a volume 
containing a target object (see fig. 4, wherein the camera being considered for 2D or 3D images); 
locating a number of features in the captured image of the target object (see page 276 and col. 1, 
as noted above); and determining an object space-to-camera space transformation for the target 
object (see fig. 4, wherein automatically the camera determines the target) based at least in part 
on a position of at least some of the located features using only the captured image and an 
algorithm (see page 276, col. 1 and page 279, col. 2, first paragraph, wherein the user of neural 
network avoids computations) that employs a known or determinable physical relationship 
between at least some of the located features (see page 277, cols. 1-2, particularly table I, 
wherein roll, pitch and yaw have been considered as physical features). 

As per claim 51, Wei et al. , teaches a multisensory visual servoing by a neural network 
including a method that further comprising determining a camera space-to-tool space 
transformation based (see page 276, col. 1, particularly "tracking an object on a conveyor" has 
automatically determine camera space-to-tool and page 277, section B. Dealing with the 
Inexactness of the Mapping has dealt with the transformation) at least in part on at least two of 
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the number of images captured by the camera of the calibration object (see fig. 4, has been 
broadly interpreted as the two images of the calibration). 

As per claim 52, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a method that comprising: determining a position of the object (see fig. 4 and table II) 
frame in the tool space reference frame (see page 2, col. 1, wherein reference frame has been 
considered as position frame); and providing an object frame to the robot (see fig. 3-4). 

As per claim 53, McGee et al. teaches a single camera for automatically determining the 
position and orientation of an object by utilizing as few as a single digital image generated by as 
few as a single camera without the use of a structured light. The digital image contains at least 
three non-colinear geometric features of the object that equates to the single camera of this 
application (see abs. and fig. 1 , particularly the single camera and "a programmed computer 
together with reference data and camera calibration data to provide at least three non-parallel 3-D 
lines. The 3-D lines are utilized by an iterative algorithm to obtain data relating to the position 
and orientation of the object in 3-D space."). McGee et al. does not specifically teach the 
following limitations as taught by the secondary reference Wei et al. 

Wei et al, in the other hand teaches a multisensory visual servoing by a neural network 
including a system that useful in robotics the apparatus comprising capture at a number of 
images of a calibration object (see abs. and fig. 4) means for calibrating the camera, by: 
determining a set of intrinsic parameters (page 276, col. 2, section II., and col. 1, second 
paragraph, wherein both intrinsic and extrinsic parameters have been covered by "sensor/camera 
calibration", and fig. 2 for motion parameters) of the camera from at least one of the number of 
images of the calibration object captured by the camera (see abs. and figs. 2-3 as noted above); 
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and determining a set of extrinsic parameters (see page 276, col. 1, second paragraph, wherein 
both intrinsic and extrinsic parameters have been covered) of the camera from at least one of the 
number of images of the calibration object (see page 276 as noted above) captured by the 
camera, the set of extrinsic parameters (see page 276, col. 2, section II., and col. 1, second 
paragraph, wherein both intrinsic and extrinsic parameters have been covered by "sensor/camera 
calibration", and fig. 2 for motion parameters) comprising a camera space-to-training space 
transformation defining a transformation between a camera space reference frame and a training 
space reference frame (see page 2, particularly "space-to-training space transformation" has been 
interpreted as from object to the camera/range finders, to the training and to the geometric 
transformation, "a space reference frame" has been interpreted as from the reference position to 
the algorithm module to the camera/range finders, to the shift and finally to the geometric 
transformation and col. 1, wherein reference frame has been considered as position frame); and 
means for estimating a pose of a target object (see page 276, section 2., wherein pose 
determination has been considered for estimation of the pose), by capturing a two-dimensional 
image of a volume containing a target object (see fig. 2-4, wherein image taken by the camera 
being considered has having the capacity to do two- or three-dimensional images); and locating 
at least six features in the captured image of the target object (see page 280, particularly table II, 
has been considered having a plurality of selectable image position); and determining an object 
space-to-camera space transformation based on at least in part on a position of at least some of 
the located features in solely the captured image using an algorithm that employs a known or 
determinable physical relationship between at least some of the located features (see page 277, 
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cols. 1-2, particularly table I, wherein roll, pitch and yaw have been considered as physical 
features and second B. Dealing with the Inexactness of the Mapping). 

As per claim 54, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a system that comprising: means for training, comprising: 

capturing an image of a teaching object (see fig. 4) of a type of object (see fig. 4) that will be 
manipulated by the robot; selecting a number of features (see page 2, col. 1) from the captured 
image of the teaching object (see fig. 4); determining a set of object (see fig. 4) space coordinates 
for each of the selected features (see page 2, col. 1) from the captured image of the teaching 
object (see fig. 4); an determining an object (sec fig. 4) spacc-to-camcra (sec figs. 2-3) space 
transformation defining a transformation between an object (see fig. 4) space reference frame 
(see page 2, col. 1, wherein reference frame has been considered as position frame) and the 
camera (see figs. 2-3) space reference frame (see page 2, col. 1, wherein reference frame has 
been considered as position frame). 

As per claim 55, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a system wherein the means for calibrating, the means for estimating a pose, and the 
means for training comprises at least one programmed computer (see abstract, page 1 , col. 1 and 
page 4, col. 2, second paragraph). 

As per claim 56, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a system wherein the means for calibrating, the means for estimating a pose, and the 
means for training comprises at least one computer-readable medium (see abstract, page 1, col. 1 
and page 4, col. 2, second paragraph) storing instructions operating at least one computer (see 
abstract, page 1, col. 1 and page 4, col. 2, second paragraph). 
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As per claim 57, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a system wherein the pose estimating means estimates the pose of the target object (see 
fig. 4) further by: adjusting a position of the movable portion of the robot (see fig. 3) if the 
number of features (see page 2, col. 1) located in the captured image of the target object (see fig. 
4) is determined to be an insufficient number of features (see page 2, col. 1). 

As per claim 58, McGee et al. teaches a single camera for automatically determining the 
position and orientation of an object by utilizing as few as a single digital image generated by as 
few as a single camera without the use of a structured light. The digital image contains at least 
three non-colinear geometric features of the object that equates to the single camera of this 
application (see abs. and fig. 1 , particularly the single camera and "a programmed computer 
together with reference data and camera calibration data to provide at least three non-parallel 3-D 
lines. The 3-D lines are utilized by an iterative algorithm to obtain data relating to the position 
and orientation of the object in 3-D space."). McGee et al. does not specifically teach the 
following limitations as taught by the secondary reference Wei et al. 

Wei et al, in the other hand teaches a multisensory visual servoing by a neural network, 
including a system that useful in robotics, the apparatus that capture a number of images of a 
calibration object means for calibrating the camera, by: determining a set of intrinsic parameters 
(see page 276 and footnote as noted above) of the camera from at least one of the number of 
images of the calibration object captured by the camera (see figs. 2-4 as noted above); and 
determining a set of extrinsic parameters (see page 276 as noted above) of the camera from at 
least one of the number of images of the calibration object captured by the camera (see abstract 
and fig. 4 as noted above), the set of extrinsic parameters (see page 276 as noted above) 
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comprising a camera space-to-training space transformation defining a transformation between a 
camera space reference frame and a training space reference frame (see page 2, particularly 
"space-to-training space transformation" has been interpreted as from object to the camera/range 
finders, to the training and to the geometric transformation, "a space reference frame" has been 
interpreted as from the reference position to the algorithm module to the camera/range finders, to 
the shift and finally to the geometric transformation and col. 1, wherein reference frame has been 
considered as position frame); and means for estimating a pose of a target object (seepage 276, 
col. 2, section II., wherein pose determination has been taken as estimation of a pose), by 
capturing a two-dimensional image of a volume containing a target object (see fig. 4); locating at 
least five features in the captured image of the target object (see table II as noted above); and 
determining an object space-to-camera space transformation based at least in part on a position 
of at least some of the located features using the captured image without any additional captured 
images and an algorithm (see page 276, col. 1 and page 279, col. 2, first paragraph, wherein the 
user of neural network avoids computations) that employs a known or determinable physical 
relationship between at least some of the located features (see page 277, cols. 1-2, particularly 
table I, wherein roll, pitch and yaw have been considered as physical features and second B. 
Dealing with the Inexactness of the Mapping). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the robot type of McGee et al, with the robot type of Wei et al., 
because this modification would have a single camera for performing the calibration into Wei's 
et al, thereby improving the efficiency and the reliability of the single camera of the single 
image 3D vision guided robotics. 
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As per claim 59, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a system wherein the means for calibrating and the means for estimating a pose 
comprises at least one programmed computer (see abstract, page 276, col. 2, section II., wherein 
pose determination has been taken as estimation of a pose and page 279, col. 2, second 
paragraph, particularly "If we use analytic computer vision methods to solve the same problem, a 
lot of calibrations have to be involved"). 

As per claim 60, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a system wherein the means for calibrating and the means for estimating a pose 
comprises at least one computer-readable medium (see abstract, page 276, col. 1 and page 279, 
col. 2, second paragraph) storing instructions operating at least one computer (see abstract, page 
276, col. 1, wherein "traditional computer" inherently works with computer-readable medium, 
and page 4, col. 2, second paragraph). 

As per claim 53, Wei et al. , teaches a multisensory visual servoing by a neural network, 
including a system wherein the pose estimating means estimates the pose of the target object (see 
fig. 4) further by: adjusting a position of the movable portion of the robot (see fig. 3) if the 
number of features (see page 2, col. 1) located in the captured image of the target object (see fig. 
4) is determined to be an insufficient number of features (see page 2, col. 1). 
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Allowable Subject Matter 

6. Claim 39 and is objected to as being dependent upon a rejected base claim, but would be 
allowable if rewritten in independent form including all of the limitations of the base claim and 
any intervening claims. 

7. The following is a statement of reasons for the indication of allowable subject matter: 
The prior art of record fail to teach determining at least one of a focal length, a first order 

radial lens distortion coefficient, a set of coordinates of a center of a radial lens distortion, or a 
scale factor indicative of a framegrabber scanline resampling uncertainty. 

8. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to MCDIEUNEL MARC whose telephone number is (571)272- 
6964. The examiner can normally be reached on 6:30-5:00 Mon-Thu. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Khoi Tran can be reached on (571) 272-6919. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 
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